1. Field of the Invention
The present invention is related to a system and method for real-time parallel delivery of segments of large payload files. In particular, the invention provides an efficient architecture and technique for storing and transmitting files containing, for example, media content such as audio, video and/or other data to a remote user.
2. Background Art
Advances in telecommunications network communication and switching are moving ahead with great speed. However, transferring large files (e.g., media content such as audio and/or video files) between locations remains a task that can take significant amounts of time. For example, when delivering a large media file, such as a movie, to a user, the transfer time can be on the order of many minutes to hours before use of the file (e.g., viewing of the movie) can begin.
One conventional technique for avoiding such delays in presenting media content to a user is to replicate copies of the content at various locations in the network “near” the user. While this may solve the delay problem, it is an expensive solution since it requires an inordinate amount of storage space to store multiple copies of the media file. It also requires a media management application for operators to manage the replicated copies of content. Storage administrators must carefully manage the storage resources to assure that no site become full. If a site runs out of storage, an attempt to replicate new content to the site will fail.
Other known techniques include application level proxy caching such as web caching and streaming caching. Such solutions do not require the deployment of unmanageable amounts of storage. However, the solution only works if the desired content is cached at a location near the requesting user. If a user requests content that is not cached, the content has to be retrieved from another location, often with an unacceptable delay.
Another problem with conventional content delivery systems is that each system processor or node must handle a number of simultaneous processes or threads. These multiple threads and processes often need atomic access to shared data stored in files. If the threads or processes are running on a single machine, one can use locking mechanisms, such as mutexes or semaphores, provided by the operating system to achieve atomicity. However, such locking mechanisms are more difficult to implement across a cluster of machines. Furthermore, file locking is not reliable when using the Network File System (NFS) to share files among a cluster of machines. One solution to the problem is to develop a lock manager to grant locks to other processes or threads. However, designing a lock manager that can handle failover is difficult. Since the lock manager must keep track of its clients, reconstructing the state before failure can be a challenge.
What is needed is a system and method for distribution of large files that overcomes the limitations of known systems.